Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: nydus support encrypted images #1333

Merged
merged 2 commits into from
Jul 6, 2023
Merged

feat: nydus support encrypted images #1333

merged 2 commits into from
Jul 6, 2023

Conversation

taoohong
Copy link
Contributor

@taoohong taoohong commented Jun 20, 2023 •
edited
Loading

Extend native nydus v6 to support handling encrypted containers images:

  • An encrypted nydus image is composed of encrypted bootstrap and chunk-level encrypted data blobs. The bootstrap is encrypted by the Ocicrypt and the data blobs are encrypted by aes-128-xts with randomly generated key and iv at chunk-level.
  • The chunk encryption key and iv are stored in the blob info reusing some items of the structure to save reserved space.
  • For every data blob, all the chunk data, conpression context. table and compression context table header are encrypted.
  • Encrypted chunk data will be decrypted first, and then be decompressed while bing fetched by the storage backend.

Related issue

@taoohong taoohong requested a review from a team as a code owner June 20, 2023 06:04
@taoohong taoohong requested review from jiangliu, changweige and adamqqqplay and removed request for a team June 20, 2023 06:04
@anolis-bot
Copy link
Collaborator

@taoohong , a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/79505

@taoohong taoohong mentioned this pull request Jun 20, 2023
Closed
5 tasks
@codecov
Copy link

codecov bot commented Jun 20, 2023 •
edited
Loading

Codecov Report

Merging #1333 (0314971) into master (c1247fd) will increase coverage by 0.00%.
The diff coverage is 38.61%.

Impacted file tree graph

@@           Coverage Diff            @@
##           master    #1333    +/-   ##
========================================
  Coverage   45.70%   45.71%            
========================================
  Files         123      122     -1     
  Lines       37287    37681   +394     
  Branches    37287    37681   +394     
========================================
+ Hits        17043    17224   +181     
- Misses      19347    19546   +199     
- Partials      897      911    +14
Impacted Files Coverage Δ
builder/src/compact.rs 0.00% <0.00%> (ø)
builder/src/core/blob.rs 13.63% <0.00%> (-0.45%) ⬇️
builder/src/core/v6.rs 32.33% <0.00%> (-0.12%) ⬇️
builder/src/merge.rs 0.00% <0.00%> (ø)
builder/src/stargz.rs 0.00% <0.00%> (ø)
rafs/src/metadata/mod.rs 52.28% <0.00%> (-1.54%) ⬇️
src/bin/nydus-image/inspect.rs 13.18% <0.00%> (-0.05%) ⬇️
storage/src/cache/dummycache.rs 21.23% <0.00%> (-0.45%) ⬇️
storage/src/cache/filecache/mod.rs 66.54% <0.00%> (-1.22%) ⬇️
storage/src/lib.rs 28.57% <ø> (ø)
... and 14 more

... and 4 files with indirect coverage changes

@anolis-bot
Copy link
Collaborator

@taoohong , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd❌ FAIL

Sorry, your test job failed. Please get the details in the link.

@anolis-bot
Copy link
Collaborator

@taoohong , the title has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/79511

@anolis-bot
Copy link
Collaborator

@taoohong , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd❌ FAIL

Sorry, your test job failed. Please get the details in the link.

jiangliu
jiangliu reviewed Jun 20, 2023
View reviewed changes
utils/src/crypt.rs Outdated Show resolved Hide resolved
jiangliu
jiangliu reviewed Jun 20, 2023
View reviewed changes
builder/src/core/blob.rs Outdated Show resolved Hide resolved
@anolis-bot
Copy link
Collaborator

@taoohong , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/79673

@anolis-bot
Copy link
Collaborator

@taoohong , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@anolis-bot
Copy link
Collaborator

@taoohong , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux❌ FAIL

Sorry, your test job failed. Please get the details in the link.

@bergwolf
Copy link
Member

bergwolf commented Jul 3, 2023

The bootstrap is encrypted by the Ocicrypt and the data blobs are encrypted by aes-128-xts with randomly generated key and iv at chunk-level.

What is iv? And I thought we agreed on using convergent encryption for shared chunk data encryption. Any reason why it got dropped?

@taoohong
Copy link
Contributor Author

taoohong commented Jul 3, 2023

The bootstrap is encrypted by the Ocicrypt and the data blobs are encrypted by aes-128-xts with randomly generated key and iv at chunk-level.

The IV is used to ensure that the same plaintext block encrypted with the same data encryption key produces a different ciphertext block each time.

The bootstrap is encrypted by the Ocicrypt and the data blobs are encrypted by aes-128-xts with randomly generated key and iv at chunk-level.

What is iv? And I thought we agreed on using convergent encryption for shared chunk data encryption. Any reason why it got dropped?

The IV is used to ensure that the same plaintext block encrypted with the same data encryption key produces a different ciphertext block each time. It's usually suggested to be provided when using AES in xts mode.
About Convergent encryption you mean CipherContext with convergent_encryption set true? It's not really dropped, the related codes are moved to utils/crypt.rs to make them capable of being more commonly and easily used by storage or rafs subsystems.

bergwolf
bergwolf reviewed Jul 3, 2023
View reviewed changes
utils/src/crypt.rs Outdated Show resolved Hide resolved
utils/src/crypt.rs Outdated Show resolved Hide resolved
@anolis-bot
Copy link
Collaborator

@taoohong , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/82310

@anolis-bot
Copy link
Collaborator

@taoohong , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux❌ FAIL

Sorry, your test job failed. Please get the details in the link.

bergwolf
bergwolf previously approved these changes Jul 4, 2023
View reviewed changes
Copy link
Member

@bergwolf bergwolf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@bergwolf bergwolf dismissed their stale review July 4, 2023 03:17

just found out that there's no tests for this.

@bergwolf
Copy link
Member

bergwolf commented Jul 4, 2023

@taoohong The implementation looks good. Could you add some integration tests (in the smoke/tests/ directory) so that it is actually tested?

@taoohong
Copy link
Contributor Author

taoohong commented Jul 4, 2023

Thanks!

@taoohong The implementation looks good. Could you add some integration tests (in the smoke/tests/ directory) so that it is actually tested?

I'd like to. THX for your review 👍

@taoohong
feat: nydus support encrypted images
7103c7c 
Extend native nydus v6 to support handling encrypted
containers images:
* An encrypted nydus image is composed of encrypted
bootstrap and chunk-level encrypted data blobs. The
bootstrap is encrypted by the Ocicrypt and the data
blobs are encrypted by aes-128-xts with randomly
generated key and iv at chunk-level.
* For every data blob, all the chunk data, conpression
context. table and compression context table header
are encrypted.
* The chunk encryption key and iv are stored in the blob
info reusing some items of the structure to save reserved
space.
* Encrypted chunk data will be decrypted and then be
decompressed while be fetched by the storage backend.
* Encrypted or unencrypted blobs can be merged together.

Signed-off-by: taohong <[email protected]>
@anolis-bot
Copy link
Collaborator

@taoohong , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/82345

@taoohong
tests: add encrypt integration test
0314971 
Add image encryption test integration case to Smoke test.

Signed-off-by: taohong <[email protected]>
@anolis-bot
Copy link
Collaborator

@taoohong , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/82346

@anolis-bot
Copy link
Collaborator

@taoohong , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@anolis-bot
Copy link
Collaborator

@taoohong , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@jiangliu jiangliu merged commit 98834dd into dragonflyoss:master Jul 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jiangliu jiangliu jiangliu approved these changes

@bergwolf bergwolf bergwolf left review comments

@changweige changweige Awaiting requested review from changweige changweige is a code owner automatically assigned from dragonflyoss/nydus-reviewers

@adamqqqplay adamqqqplay Awaiting requested review from adamqqqplay adamqqqplay is a code owner automatically assigned from dragonflyoss/nydus-reviewers

Assignees
No one assigned
Labels
anolis_test_pass
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@taoohong @anolis-bot @bergwolf @jiangliu